#inference optimization

'Huawei's CloudMatrix builds a peer-to-peer supernode combining 384 Ascend 910C NPUs and 192 Kunpeng CPUs to deliver high-throughput, low-latency LLM serving, with CloudMatrix-Infer optimizing MoE and KV cache workloads.'

'A technical deep dive into AI inference in 2025, detailing latency bottlenecks, optimization methods like quantization and pruning, and a roundup of the top nine inference providers.'

NVIDIA researchers developed Dynamic Memory Sparsification (DMS), a novel method that compresses KV caches by 8× in Transformer-based LLMs, improving inference efficiency while maintaining accuracy.